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ABSTRACT 


Various NASA Langley Research Center and other center projects were attempted for 
analysis to obtain historical data comparing pre-phase A study and the final outcome for each 
project. This attempt, however, was abandoned once it became clear that very little 
documentation was available. Next, extensive literature search was conducted on the role of risk 
and reliability concepts in project management. Probabilistic risk assessment (PRA) techniques 
are being used with increasing regularity both in and outside of NASA. The value and the usage 
of PRA techniques were reviewed for large projects. It was found that both civilian and military 
branches of the space industry have traditionally refrained from using PRA, which was developed 
and expanded by nuclear industry. Although much has changed with the end of the cold war and 
the Challenger disaster, it was found that ingrained anti -PRA culture is hard to stop. Examples of 
skepticism against the use of risk management and assessment techniques were found both in the 
literature and in conversations with some technical staff. Program and project managers need to 
be convinced that the applicability and use of risk management and risk assessment techniques is 
much broader than just in the traditional safety-related areas of application. The time has come to 
begin to uniformly apply these techniques. The whole idea of risk-based system can maximize the 
‘return on investment’ that the public demands. Also, It would be very useful if all project 
documents of NASA Langley Research Center, pre-phase A through final report, are carefully 
stored in a central repository preferably in electronic format. 
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INTRODUCTION 


It is desired to control projects by effective application of risk analysis at the planning stage. 
Such control is even more desirable if projects are exposed to highly uncertain environments. 
NASA projects are possibly the best examples of projects subject to great uncertainty not only 
because there is often no precedence, but also some projects involve harsh physical operating 
environment of Space. Risk identification is a period of intense interaction among all parties 
involved: the analysts, project team, and management. The nature of all possible risks and the 
manner each risk may arise should be discussed. It is important not to limit the concepts of 
“project reliability” and “project risk assessment” to more common probabilistic project analysis 
concepts often centered on the PERT and other similar techniques. The probabilistic nature of 
project completion time is just one aspect of the outcomes and/or ingredients of this study. 

The importance and urgency of risk analysis in today’s complex projects, in face of financial 
constraints, has spurred several research efforts in this area. Cost overruns are commonplace. 
One major reason for cost overruns is the uncertainty inherent in various aspects of the work. 
This uncertainty can result in a wide range of outcomes that in turn may impact project cost and 
schedule in unfavorable ways. Risk assessment is difficult in large projects. Yet, it is imperative 
that the owners or sponsors engage in a rigorous, systematic analysis of major sources of risk. 
Risk, as used in the context of this report, is defined primarily as the potential for monetary loss 
resulting from uncertainty about the project. In order to develop the risk management 
framework, first the sources of risk must be identified and categorized. Then a measurement 
system should be used to quantify the risk. Finally, each risk item should be allocated between the 
parties involved in an equitable manner If the project risks can be identified in a timely manner, 
quantified in a logical way, and allocated properly between the project participants, then the 
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likelihood of significant cost and schedule overruns will be reduced considerably. Many 
parameters may be responsible for budget overruns. Scope changes or optimistic scenarios 
yielding low estimates of costs and high estimates of benefits, incomplete information about the 
project objectives and features, estimation error, and delay in start date are some of the more 
important parameters contributing to the budget overruns. Some of these factors are of a 
technical nature and depend on the project complexity, location and size; others are financial 
issues and are affected by the economy, affordability, cost of funds, and the owner s 
creditworthiness. Still, other factors depend on the political atmosphere surrounding the decision- 
makers and the general public. Although these social and political factors are of utmost 
importance, they are not the primary subjects of this report. The first step in a risk management 

program is to identify risk prone areas in a project. After the risk identification process, a 
methodology for measuring design, construction and financial risks should be devised. The 
methodology, though based on sound theoretical principles, must be practicable and convenient to 
apply to real life problems. After risks are appropriately identified and measured, they should be 
allocated to various parties involved in the project in a fair and equitable way. This should be 
done in a way that ensures the prudent expenditure of public funds. Every technique for risk 
analysis must begin with the development of a method for the identification and classification of 
individual risks inherent in a particular project. While every project has its own unique set of 
risks, there are many risks that are common to all projects. Examples include unknown 
conditions, severe weather possibilities, contractor reliability, and the risk of maintaining adequate 
funding. One of the most adaptable methods for risk identification and classification is the 
development of a risk checklist. This technique allows the user to list common project risks, and 
then to append the list with those risks peculiar to the project at hand. Risk identification is 
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heavily dependent upon the experience and perceptivity of project management. In order for a 
checklist to be effective, there must be a concentrated effort during the development stage to 
identify all relevant risks by all members of the management team. This process can be 
particularly arduous because humans are not predisposed to identify more risks and thereby 
creating more things to worry about. By identifying risks and developing appropriate courses of 
action should such events occur, management will transcend the “putting out fires” mode. That 
is, management will become proactive instead of reactive. 

There are several different approaches to organize a risk checklist into a logical, 
understandable, and useable format. One approach proposes that risks should be organized in 
terms of the nature of the risk itself. Specifically, risks can be classified as either knowns, known- 
unknowns, or unknown-unknowns. A known risk is an item or condition that is understood, but 
cannot be measured with complete accuracy. Generally, such risks occur at a relatively high rate 
and contain a range of possible outcomes. Labor productivity is a good example of a known risk. 
Known-unknowns conditions or events that are foreseeable, but not normally expected. A 
second method for organizing a risk checklist is to classify the risks according to their nature and 
their primary sources. Under this scenario, risks are placed into one of the following categories: 
external-unpredictable, external-predictable, internal non-technical, technical, and legal. Examples 
of external-unpredictable risks include natural hazards or regulatory changes. External- 
predictable risks involve inflation, currency changes, environmental impacts, and social impacts. 
Internal, non-technical risks are embodied by items such as schedule, cost, cash flow, and 
management. Technical risks evolve from changes in technology, from sheer size or complexity 
of the project, and from design or performance standards. Finally, legal risks arise from patent 
rights, licensing, contractual problems, and insider and outsider lawsuits. This classification 
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system provides the benefit of arranging the groups according to their relative controllability. For 
instance, natural hazards are considered external-unpredictable and have a low degree of 
controllability while contractual risks are ranked as legal risks with the highest controllability. 
Yet another approach to classifying risks is based upon their effect on the project. Under this 
method, risks would be considered as either cost risks, schedule risks, or quality risks. 
Unfortunately, many risks fall into more than one category, and accordingly, create the potential 
for double counting when mitigation procedures are being considered. Almost every party 
involved in the project needs to perform its own kind of risk analysis. While the owner has to 
look at risk issues at a more macro or aggregate level, the contractor would be wise to consider 
chance variations at a more detailed level. The owner, public or private, needs to assess the 
amount of uncertainty in the project cost and schedule in order to make plans for seeking project 
funding. Multi-year megaprojects are particularly sensitive to variations in project duration. The 
cost of money needed to finance these projects become prohibitively high as the project duration 
increases. Because of these issues, financial risks become of paramount importance to the owner. 
If the sponsor is the Federal government, legislative issues such as funding authorization and 
appropriation have to be considered also. Sources of funding and its composition, the 
commitment and reliability of local sources, the accuracy of estimating funding levels over project 
life, and the probability of project failure due to optimistic assumptions all add to the project’s 
financial risks. The owner should also concern itself with the contractor selection process, the 
stability and strength of the contractor in executing a large transit project, and expected loss levels 
in case the contractor fails to complete the project. Even if the contractor does not default, the 
owner or the sponsor has to evaluate the probability and the potential loss in the case of project 
delay and cost overrun. 
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The traditional contractor on the other hand, looks at a project’s risks from a different angle. 
Although financial risks are very important and the contractor would want to be sure that the 
owner has sufficient funding to finance the project, contractor will be concerned with the amount 
of funding that would be needed for interim financing. Interim financing fills the gap between the 
contractor’s spending and income in a project. The smaller this gap, the less expensive it would 
be to finance the difference between the contractor’s expenditures and progress payments. The 
cost of interim financing cuts through the contractor’s profit margin and because of this the 
contractor should carefully study the expected levels of needed financing. Also the contractor 
needs to pinpoint areas of risk and uncertainty in the project and assess the impact of those areas 
on the project cost and duration in order to include a reasonable contingency in the bid, especially 
in competitive lumpsum contracts. Careful evaluation of this contingency is important. A low 
estimate of the required contingency may get the contractor the job but may cost him dearly after 
the project starts as the time and cost variations may develop an unfavorable impact on the 
project. A high or conservative estimate of contingency on the other hand, will put the contractor 
at a disadvantage because his bid may not be competitive enough to get the job. There are two 
general approaches to evaluation of variations of project components. Some approaches are 
based on some deterministic safety margin for critical items based on expertise of the seasoned 
personnel or historical data compiled from similar projects. In some cases these deterministic 
methods tend to work well because of the nature of the available data and the experience of the 
analysts. For example, in many cases a well-designed sensitivity analysis is all that is needed for 
assessing the risk impact on a project. Other approaches are based on some probabilistic model 
where the variability of important parameters are formally introduced into the predictive models. 
With the recent developments in risk analysis software and the increasing familiarity of engineers 
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and analysts with probabilistic approach, these methods are being used much more extensively. 
The probabilistic method provide the user with much more information compared to deterministic 
method and helps the user make informed decisions. In the deterministic approach, the potential 
cost overrun for the project is estimated based on the experience of the personnel and all the 
information that can be obtained from similar projects and the project under study. It is common 
to see a contingency rate of around 10% added to the total project cost in order to cope with 
project uncertainties. This approach, especially if taken by the owner can lead to problematic 
results 

There are several reasons for the owner to calculate contingency using a systematic approach 
to risk identification and assessment. Many times the contingency rate is added arbitrarily and not 
without elaborate analysis. Also, some risk elements are counted twice as they have been 
considered in the estimating phase. Adding an overall contingency rate only considers the 
potential for loss as it increases the project costs. It many cases though, the probability of 
underrunning certain cost elements is reasonably large and has to be incorporated into the model. 
Furthermore, often it is not clear that the contingency gives the expected value of cost overrun, 
the most likely value of the cost overrun, or the worst case scenario for the project cost. The 
likelihood of arriving at a certain project budget cannot be assessed with this method. Even if its 
definition is clearly given, still the owner may not be able to decide on the actual level of reserve 
funds. For example, is it reasonable to provide for the worst possible scenario and hence possibly 
jeopardize project’s viability when the probability of realizing such a cost is extremely low? A 
more reasonable approach is to identify major risk elements in the project and assign reasonable 
contingency rates to these various items. These contingency rates may not be the same from area 
to area. The total contingency budget will be the sum of the products of the individual 
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contingency rates and respective component estimates. This approach has the added benefit of 
earmarking contingency budget for various project components. This will allow for a more 
efficient contingency drawdown policy and can alert the management if a certain component is 
using too much of the reserve funds. In these approaches it is important that costs be estimated 
as realistically as possible. In other words, based on the information at the time of preparing the 
estimate a fair cost of the component should be calculated without trying to safeguard against risk 
elements. The impact of uncertainty shall then be considered when arriving at the contingency 
rates by carefully evaluating the risk checklist and drawing upon the experience of the people 
involved in the project and historical data from similar jobs. 

Project cost and schedule are interrelated. Given the shear size of some projects and large 
amounts of funds required, project delays drive up the cost of money drastically. Setting realistic 
objectives for project milestones and the completion date is one of the first steps in calculating the 
project financial needs. The project financial needs in turn impact the budget and the cost 
contingency. A logical approach in schedule risk analysis is to refer to a carefully developed CPM 
schedule. Through the CPM one will be able to see the interrelationships between various 
elements of the project and to evaluate the impact of an activity delay on various milestones and 
the completion date. The schedule for the owner/sponsor will be different from the contractor’s 
schedule in that it will encompass planning and design phases in addition to the construction 
phase. Reasonable contingencies can be built into project schedule in terms of floats for various 
milestones. The larger the amount of these floats and the smaller the number of milestones that 
carry liquidated damages clauses, the less risky the project from the constructor point of view. 
Including stiff liquidated damages in a tight schedule with several milestones will result in bids 
with high contingencies. An important benefit of using, CPM schedule is that it ranks activities 


10 



(or the project components) according to their impact on project milestones and the final 
completion time. The activities that have higher floats are less likely to create schedule delays. A 
deterministic risk analysis can at best provide an upper limit and/or a most likely value (or in some 
cases an expected value) for the risk of performing a project. The user will not have information 
about the likelihood of needing a certain level of contingency. The importance of relating various 
levels of exposure (or contingency) with probability of their realization cannot be 
overemphasized. Without knowledge of this relationship, the effectiveness of decision making 
will become random. On the other hand, if uncertainty of various variables are formally 
introduced into the cost and schedule models, then one can arrive at a distribution for the 
outcome of the analysis. This distribution allows the analyst or the decision-maker to make 
informed decisions regarding the project’s management, budget and schedule. Indeed, many may 
suggest that there is no such thing as “deterministic risk analysis” because risk by definition is 
derived from uncertainty, which in turn is a probabilistic concept. 

Implementing a probabilistic approach in risk assessment is generally more complex than the 
traditional deterministic approaches and requires more input data. Conveying the results of a 
probabilistic approach to the top decision-makers may be more difficult as well. Despite these 
issues, effort should be made that a probabilistic analysis be conducted to assess the levels of risk 
in a project. Without a probabilistic approach a complete profile of project risks cannot be 
developed. In general, the probabilistic approach in assessing risk or measuring probability of 
cost or schedule overrun/underrun is to treat various components of the project, especially those 
components that are expected to vary greatly, as random variables. The underlying assumptions 
in both probabilistic scheduling and estimating are so similar that one can discuss both subjects at 
the same time In almost every case, a model is developed for predicting the project cost or 
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schedule. As this model is a function of several random variables (those components of cost or 
schedule that have a fair chance of variation and are expected to contribute to the total project 
uncertainty), it is itself a random variable. If one can estimate the distribution of the random 
variable that is used to model total project cost or total project duration, then one can compute 
probabilities associated with various levels of confidence regarding meeting a specific deadline or 
a prescribed budget level. The problem is that in many cases it would be very difficult if at all 
possible, to analytically find the distribution of the random variable representing total project cost 
or schedule. That is why in many cases a simulation analysis is conducted to arrive at the 
Cumulative Distribution Function (CDF) of the total cost or schedule. The following factors may 
affect the analysis outcome: 

• The choice of statistical distributions and parameters used to model individual project 
components 

• The choice of the mathematical model for the total project cost or schedule 

• The choice of analytical technique used to solve the predictive model 

As mentioned earlier, the general approach in assessing uncertainty in construction projects is 
to treat project components with a high potential for variability as random variables. So an 
activity’s duration traditionally estimated with a single number, or a unit cost item that the 
estimator usually estimates based on the information available deterministically, are modeled as 
random variables with specified means and variances. In most cases, specification of a 
distribution type is also needed in order to be able to conduct a probabilistic analysis. Almost 
always, a well-known theoretical statistical distribution is used to model the item’s variability. 
This is due to the fact that these statistical distributions are well known, usually fully documented, 
and therefore easier to work with and to evaluate. Given the variety of statistical distributions 
available, one is generally able to choose a reasonable distribution for modeling a certain 
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parameter’s variability Regarding financial risks, one of the most important items is the interest 
rate used in the analysis. Interest rate is a function of the inflation rate, economic growth, and 
loan duration. Both inflation and economic growth can be closely modeled by a normal 
distribution. The additional premium associated with loan duration may be modeled as a linear 
function of time. So the interest rate can also be modeled as a distribution. 

The most common approach in probabilistic scheduling is PERT where every activity is 
modeled as a random variable distributed according to a beta distribution. The total project 
duration is computed along the network’s critical path (the longest path) by adding the means of 
the activities on the critical path. According to Central Limit Theorem (CLT), the sum of several 
independent and identical random variables is a random variable with an approximately normal 
distribution. The mean of this normal random variable is the sum of the means of the individual 
random variables and the variance of the total is the sum of the variances of the individual random 
variables. In this way, the total project duration is modeled as a normal distribution and its 
parameters can be conveniently estimated from the activity data. If activity durations are not 
independent then the use of Central Limit Theorem is not theoretically justified. The other 
concern in applying CLT to PERT is that in some cases, several paths in the project are almost as 
long as the critical path. In these cases it is possible that the shorter paths that happen to have 
larger variances than critical path will become critical. In such cases, the question is to what path 
the CLT should be applied and which path is actually going to be the longest? One suggested 
solution has been to use the Monte Carlo simulation in analyzing these cases. This issue has been 
discussed under merge event bias problem in various publications. In the Monte Carlo simulation 
approach, a random number is generated on a computer to generate a duration for each activity 
using its distribution. These numbers are used to schedule the network and the total project 
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duration is computed. In this process the activities on the critical path (the sequence of activities 
with the longest total duration) are identified. This process of generating random numbers 
according to various activity distributions is repeated many times (from several hundred times to 
several thousand times) and every time the critical activities are identified. Then a criticality index 
is computed for each activity that reflects the probability of the specified activity becoming 
critical. This criticality index is simply the ratio of the number of times an activity was on the 
critical path to the total number of simulation runs. In this way, the activities with a high 
probability of becoming critical are identified. This can help the management to allocate a proper 
level of attention to these components of the project. 

The analyst has the option of using either a general-purpose simulation language such as 
SLAM or SEMAN to develop a model of the project schedule, or use a specially designed 
software package that allows conducting Monte Carlo simulation on a scheduling network. The 
first approach is much more flexible but requires more time and the user has to have expertise in 
modeling probabilistic systems. In such an approach, risk measurement can be done either using 
traditional network-based schedules or utilizing any appropriate relationship that realistically 
defines a duration or productivity rate. Using a CPM schedule has the advantage that depicts 
activity precedence and can serve as a convenient environment for developing a schedule risk 
study. The traditional network lacks the flexibility needed in modeling complex yet quite probable 
situations. One such flexibility is the possibility of probabilistic branching. As an example, 
consider a project where the source of local funding is uncertain. Maybe the local agency or the 
owner is not sure if the public is ready to foot the bill required for the local contribution. In 
developing a schedule for the project, it would be wise to consider two paths. Each path has a 
certain probability of realization. For example, the analyst may think that there is a 75% 
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probability that the public will support a new tax to pay for the local share. There is a 25% 
probability however, that the proposed tax will not be accepted and this can direct the project 
schedule through a loop consisting of several activities (further negotiations, study, etc.) with a 
duration of several months. If the network can be modeled such that it allows probabilistic 
branching after every milestone, this uncertainty can be incorporated into the model and proper 
actions anticipated. Other potentially useful information would include but not be limited to 
activity criticality indices, the distribution of time between any two milestones in the network and 
flexibility in modeling correlations between activities. The second and easier option is to use a 
software package specifically designed to perform Monte Carlo simulation on a CPM network. 
Because of the increasing interest in probabilistic scheduling, software, companies have developed 
such computer programs. In one such example , the software allows the user to. either define an 
empirical distribution for an activity or choose from a number of distributions (triangular, 
negative exponential, empirical) for modeling activity duration times. The software allows the 
user to model activity correlations by using the same percentile values when sampling from 
correlated distributions. This assumption reduces the system’s flexibility somehow, but is an 
improvement over the assumption of independence that PERT uses. The software also permits 
probabilistic branching. It is expected that many more software developers will market software 
in this area in the near future. Many factors affect the choice of methodology in network 
analysis. This information can help in assessing the impact of this module on other construction 
packages in this transit project. Depending on the Master Schedule for the project, if the module 
studied here is on the critical path and can cause delay in the final project completion time, then it 
would be wise to study alternatives for schedule compression. A common application of risk 
analysis in construction is to compute the CDF of the total project cost. This in turn can help the 
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owner specify margins of safety needed for the levels of funding required. The CDF can help 
arrive at a reasonable contingency sum and to allocate contingency to various project activities. 
Again Monte Carlo simulation technique is continuously used in cost risk assessment. At this 
point we will examine the typical cost functions that are used for risk modeling. 

Obviously, if one wants to consider cost variations in every small cost component that goes 
into a detailed estimate, the approach would be impractical. Also, it is understood that most of 
the total cost variation is due to the variability of a limited number of components. So only those 
items with high potential for variation are considered as random variables and the rest of the items 
are assumed to be fixed. Any single component that has the potential of changing the project 
bottom line by more than this critical variance is considered a critical component and should be 
modeled as a random variable. Monte Carlo simulation can simplify the process if a computer and 
the relevant software are available. It consists of generating random numbers according to q 
distributions, adding up these items, adding the fixed costs to these, and computing the total 
project cost. This procedure is repeated at least several hundred times, and every time a value for 
total cost is computed. The number of iterations needed depends on the complexity of the model 
and how quickly the results of the analysis converge. It should be chosen sufficiently large so that 
the outcome of the analysis does not change by further increasing the number of iterations. 
Although the Monte Carlo approach provides a straight forward means for probabilistic 
estimating, there are major limitations in its application. First, one needs to establish statistical 
distributions for various cost components. Second, if the random numbers are not independent, 
their correlations should be fully documented for the correct implementation of the Monte Carlo 
technique. Underlying Statistical Distributions: One logical method for investigating the 
distribution type is to collect data from similar projects, assume a distribution, and perform a 
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proper test of goodness of fit to evaluate the hypothesis. In the absence of historical data, the 
same general guidelines regarding the choice of distribution mentioned earlier in the report can be 
used. Correlation between project cost components: One of the more common sources of error 
in Monte Carlo simulation is that it is assumed that cost components are independent and changes 
in one cost component do not affect any other cost component. This is clearly inaccurate in 
typical NASA projects; however, it is assumed that if the correlation between variables is 
sufficiently small, the assumption of independence does not create large errors. Generally, 
disregarding the correlation between variables in a Monte Carlo simulation results in an 
underestimation of the total cost variance as the effect of covariances (that are mostly positive) in 
computing the variance is neglected. By neglecting the effect of correlations among variables, the 
variance of the total cost can be underestimated by 50%. This is clearly an error in the unsafe 
direction as larger variances mean higher probability of cost deviation. 

An Approximate Method for Incorporating Correlations: The accurate method of 
incorporating correlations is time-consuming and requires a great deal of data that is not always 
available. In some cases, if the underlying distributions are not normal, it is not possible to make 
an accurate analysis. The Accurate Method for Incorporating Correlations: For conducting an 
accurate analysis of total cost variance, the joint density functions of the correlated cost 
components are needed. The PDF that the estimator or risk analyst specifies for a certain cost 
component is actually the marginal distribution of that cost component In general, if different cost 
components are not independent, knowing the marginals of these random variables is not 
sufficient to obtain their joint density functions. Without the joint density function, the correlated 
random numbers cannot be generated for Monte Carlo simulation. The case of multivariate 
normal distribution is an exception, however. If one has marginals of the multivariate normal 
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distribution and the covariance matrix, then one can generally find the joint density and conduct 
the analysis. This means that the cost components have to be normally distributed. Multivariate 
normal distribution can be transformed to multivariate lognormal . Also, in special cases, one can 
use approximations to analyze the correlated random variates at the cost of reduced accuracy. 

This level of detail in conducting risk analysis in construction however, is almost never attempted 
in practice and the assumption of independence or the simpler method described above is all that 
is actually used. Rank correlation coefficient between two random variables measures the 
correlation between the two random variables. Many of the software packages developed for risk 
analysis (@RISK[40], for example) allow the user to specify correlation coefficients between 
several random variables and then generate correlated random numbers. It should be noted that 
these specified correlations are rank correlations rather than the more familiar Pearson correlation 
coefficients. Although several authors have claimed that rank correlations are indeed very good 
measures for describing the degree of association between variables, but this assertion requires 
further study, especially in the domain of cost and schedule risk analysis. Again the Monte Carlo 
approach can be used to develop a CDF for total cost. Any of the parameters described above 
may have variations that have to be considered in the analysis. An analytical solution may not be 
always convenient or even feasible depending on the shape of the cost function. Computations 
become cumbersome especially if reasonably complex and realistic distributions such as lognormal 
or beta are assumed for the parameters. 

Once risks have been identified and measured, the process of risk allocation amongst the 
parties involved in the construction project may begin. Since the owner is the one who provides 
the money, it is his privilege to assign responsibilities. Accordingly, he has the opportunity to 
reduce the total project cost through effective allocation of financial, design, and construction 
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risks. Publicly funded projects are usually awarded on a lumpsum basis through competitive 
bidding. Although objectives and specific requirements of major Space transport systems may 
generally be defined carefully, not all of the project details are known in advance. A good portion 
of these contracts involve construction of facilities where future work cannot be predicted with 
great accuracy. Also, some of these projects are so complex that there are few eligible contenders 
to bid on the job. The traditional lumpsum approach where the total risk is placed on the 
contractor’s shoulders through rigid contractual language is not necessarily optimal. Experience 
has shown that it is the owner who ultimately bears the burden of risks, whether he originally 
accepts them, whether he assigns them to the contractor and receives them back in the form of 
higher bid contingencies and change orders, whether he receives no proposals because he 
transfers all risk to the contractor, or whether he pays for them via court decree. Contract 
documents should be prepared by the owner with full knowledge of management and engineering 
as to how the risks will be allocated with adequate time for the selection of the appropriate 
language, and with sufficient time for review. With reference to optimal risk allocation, there are 
several tenets which owners should follow. The primary doctrines of risk allocation are: 

• Allocate the risk to the party who is in the best position to control it 

• Which party is in the best position to accept the risk if it cannot be controlled? 

• Consider the ability of the party receiving the risk to survive the consequences if the risk 
occurs 

• Consider whether the dollar premium charged by the transferee will be acceptable and 
reasonable 

• Do not penalize a party for accepting a risk; for example, do not use a no damages for 
owner caused delay clause in conjunction with a liquidated damages clause 

• Evaluate the potential for new risks being transferred back to the owner when initial 
allocations are made 

Various experts have developed risk management strategies to help the owner select the most 
suitable option for a given risk. Since many options appear simultaneously in various references, 
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we first delineate each recommendation in a succinct form and then explain the common 
interpretation of all possible options. The references chosen here have used several references 
themselves, so the following is the result of numerous studies, projects, and individual expertise. 
In short, this synthesis conveys the state of knowledge on risk allocation at this time. Who is in 
the best position to control the events that may lead to the risk event? For example, when a new 
large airplane is proposed to fly over a densely populated urban area, vibrations from a passing 
plane are likely to impact adjacent buildings. Since the designer is in the best position to minimize 
the likelihood of these vibration, he should be allocated such a responsibility. Based on the 
foregoing studies and other extensive research, we have concluded that risks may be allocated by 
one or more of die following options: 

• Risk acceptance 

• Risk reduction 

• Risk sharing 

• Risk transfer 

• Risk avoidance 

The list has been organized such that responsibility and ultimate control that the owner retains 
for a particular risk changes from high to low. For example, if the owner accepts the risk of 
inflation, he has relieved the contractor of the risk burden altogether. He has placed himself in the 
position of controlling the inflation risk and must consider other options. At the other end of the 
spectrum, an owner may choose to avoid a risk. As a result, he will hope to have no 
responsibility for it and have little control over it (other than to continue to avoid it). These five 
options, while covering all methods of risk mitigation, consolidates some mitigation measures 
suggested by others. For example, insurance is generally considered as a risk transfer measure. 
So there is no need to have both insurance and risk transfer as independent mitigation measures; 
rather, insurance is treated as a subcategory of risk transfer. Similarly, risk acceptance with 
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contingency and risk acceptance without contingency are both methods of accepting the risk and 
can be treated under one mitigation measure. It should be noted that in many cases, a 
combination of these measures are called for to properly allocate and mitigate a certain risk. Risk 
Acceptance: Risk acceptance connotes that the owner will assume the whole or a portion of the 
monetary impact of the risk. Note that acceptance may be planned or uncontemplated. A 
planned risk acceptance indicates that, the owner has thoughtfully investigated and deliberately 
chosen to retain an identified risk. In order for a risk to be accepted it will generally comply with 
one of the following conditions: 

A. It is voluntarily assumed 

B. No alternative is available 

C. The risky outcome is unknown with certainty 

D. Exposure is essential 

E. The negative consequences are ordinary 

An uncontemplated risk acceptance occurs when the owner fails to identify or recognize the 
risk, and therefore unknowingly accepts the risk that may happen. Generally, such instances 
occur when the owner fails to perform a thorough risk identification analysis, and by default, 
passively retains the risk and this is when it is most costly to the owner Alternately, 
uncontemplated risk acceptance occurs when the owner correctly identifies a risk, but fails to or 
cannot properly assess the size of the potential losses. Risk acceptance may be made with 
contingency or without contingency. Contingency is a sum of money or period of time set aside 
from the general funds to pay for losses that actually occur. The total contingency budget will be 
the sum of the contingencies calculated for various risk components in the project. To the extent 
that total project costs do not exceed the planned budget with the planned contingency sums, the 
owner will not have to search for additional funding. Risk acceptance without contingency should 
only be considered when funding limitations preclude a properly implemented contingency 
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account. This however, is a risky strategy. If such an instance should occur, the accepted risk 
items should have a low probability of occurrence or low potential impact. Risk Reduction: In 
the context of this report, risk reduction implies that the owner has accepted the risk but has taken 
certain defensive planning actions to lower its potential impact. This may be accomplished in two 
ways: 1) lowering the probability of a risk, and/or 2) lowering the dollar impact of the risk if it 
does occur. Risk reduction may also be accomplished by selection of an alternative, which 
possesses a lower risk. The alternative may be a different process, material, or method that still 
accomplishes the same goal. Alternates are often engendered by reviews, alternative bids, and 
value engineering Risk Sharing: When it is impossible or impractical for one party to control a 
specific risk, the task may be better managed by dividing it such that two or more parties manage 
the portion that they are best able to control individually. An excellent example of risk sharing is 
the development of a joint venture by contractors. A joint venture is the result of the unification 
of two or more contracting firms to build a single project. These types of organizations are often 
extremely well suited for the pooling of complimentary resources and facilities, for spreading 
construction risks, and for accomplishing tasks greater than any individual firm acting alone can 
undertake. At a risk item level, an owner may share inflationary risks with a contractor in projects 
with long durations. In this way both parties will be exposed to a risk item none of whom have 
much control over At the contractual level, risks may be shared through the use of a Guaranteed 
Maximum Price Contract With this type of contract, the contractor is reimbursed for costs 
incurred plus a fee up to the contract ceiling. If the project costs exceed the guaranteed 
maximum, the owner is exposed to risks for the costs below the ceiling. It should be noted 
however, that cost plus contracts are not commonly used in public works contracting. Because of 
this, we will not be investigating this option in great detail. Risk Avoidance: One obvious 
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measure to avoid risks is not to proceed with the project at all. This option may not be always 
available. However, it is still possible to avoid certain risky tasks, materials, or processes. For 
example, use of a new technology, although potentially attractive, may result in costly 
complications; a traditional technology in such a case would avoid the risk of using that new 
technology altogether. As various phases of project planning and design such an Alternatives 
Analysis, Draft and Final Environmental Impact Statements are completed and approved, the 
ability to avoid risks diminishes. For example, many nuclear power propelled space transportation 
projects belong to this class. In such cases, other mitigation measures are usually used to limit 
the owner’s exposure to risk. 

Most engineering work usually involves identification of a problem or a demand and fixing 
this problem or designing a product to meet the demand. Most engineers prefer to deal with the 
absolute than with probabilities. While borrowing historical data from other designs and some 
degree of testing, and the experiences of others, entire organization may well be accustomed to 
work “test and fix” basis rather than employing more holistic, sophisticated tools like quantitative 
risk assessment or risk management (RM). Although not unique at all, NASA was such an 
organization well after the Apollo era, but this has been changing gradually. Resistance of NASA 
to RM is actually understandable according to a historical note by Bell [7, page 44] : NASA was 
told by the reliability study contractor (GM) that there was only less than 5% chance that Moon 
trip would be successful, but all seven trips (including Apollo 13) were resounding successes. It 
is reported that [7] NASA chose to build confidence by design, not by statistical test programs. 

It is important to note that NASA had much stronger political support and financial means during 
that time period. It is sometimes difficult to tell how and if RM was used even in a recent NASA 
project For example, a recent presentation[58] included much reference to RM as an integral 
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part of a successful project, but the presenter later revealed that RM was really an after thought 
add on to this project. It is hard to argue against such after the fact RM activity especially if the 
project was a success. On the other hand, Batson[5] reports numerous NASA cases of some 
degree of failure where RM should have been used. 

RM is an iterative process to identify risks, assess product/program impacts of risks, and 
develop options to manage risks. RM is maintained throughout the life of program to 
continuously identify risks and abate early to minimize program impacts. RM is an organized 
means of identifying and measuring risk (risk assessment) and developing, selecting, and 
managing options (risk analysis) for resolving (risk handling) these risks. A summary of the RM 
process based on a number of references is summarized below. 

RM process consists of: 

• Identify 

• Assess 

• Analyze 

• Abate 

• Risk Management Tools 

• Identify 

• Metrics 

• Expert interviews 

• Team members' expertise 

• Program evaluation 

• Technical baseline 

• Schedule 

• Cost estimate 

• Resources 

• External factors 

• Brainstorming 

• Assess 

• Threat definition 

• Assumptions 

• Constraints 

• Groundrules 

• Physical measurement 
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• Risk Scoring Method 

• Probability of occurrence 

• Consequence of occurrence 

• Logic networking 

• Requirements changes 

• Relative Ranking 

• Fishbone diagrams 

• Pareto Charts 
Analyze 

• Decision Analysis 

• Metrics 

• Trend analysis 

• Statistical process control 

• Variability histograms 

• Probabilistic network tools 
Abate 

• Flow diagrams 

• Abatement status and analysis 


Issues — Threats -- Risks 

• Issue - a point of debate or controversy 

• Threat - a concern perceived as having a potentially adverse impact on achieving program 
goals 

• Risk - a threat that has been assessed as having sufficient probability of occurrence and 
severity of consequences as to significantly impact the program baseline 

Categories of Risk 

• Technical - uncertainty of performance of hardware and software 

• Supportability - uncertainty pertaining to reliability, maintainability, and resupply 

• Schedule - uncertainty of program schedule and ability to accommodate internal slippage 
without impacting major program milestones 

• Cost - uncertainty of specific program element costs and program's ability to 
accommodate cost increases within the program resources 

• Programmatic - uncertainty that sometimes may be outside program control such as 
political concerns, funding, international partner relations, product group relations, labor 
disputes, as well as risks from across program elements including contracts, personnel, 
requirements stability 

• Safety - uncertainty related to the presence of hazards to the health or life of station 
assemblers, operators, or inhabitants 

Guiding Precepts of RM 

• RM is not a new task; it is a normal way of doing business. 

• Identifying threats is everyone's job. 
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• Implementation works best at lowest possible level. 

• The sole intent is to help teams produce product, not to produce risk paperwork. 

• All risks are relative; absolute risk numbers cannot be determined. 

• Risks are inter-active across and intra-active within products/functions, and cumulative 
across program elements. 

• All activities have risks, so always identify, assess, analyze, abate risks to help ensure 
meeting team objectives. 

Risk Management Process and Tools 
Step 1 : Identification 

Process of Threat Identification 

• Take personal responsibility for identifying threats 

• Look for potential threats to program goals 

• Determine the scope of the perceived threat 

Tools for Threat Identification 

• Management Emphasis System (MES) 

• Metrics 

• Expert interviews 

• IPT members' expertise 

• Program evaluation data (requirements, schedule, cost estimates, etc.) 

• Brainstorming 

• Informal means — any one, any means, any time 
Threat Identification Questions 

Technical 

• How mature is the design? 

• What proven and unproved technologies are being used? 

• How complex is the system? 

• Is the operating environment well-defined? 

• Are the interfaces well-defined? 

• How mature is the test plan? 

• What is the status of the test equipment design/fabrication? 

• What are the verification difficulties? 

• Are the necessary manufacturing facilities available? 

• Is Statistical Process Control (SPC) used to control the manufacturing process? 

• Are new processes and training required? 

• Is manufacturing satisfied with the design producibility? 

• What is the material availability? 

• Will the design produce a supportable product 9 

• Is complex tooling required? 

• What is the status of the tooling design/fabrication? 
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• How good are the analytical tools? 

Supportability 

• What are the transportation concerns? 

• Are there user/payload issues left unresolved? 

• How much return/resupply weight is required? 

• How much on-orbit time is required for system maintenance? 

• How reliable is the system? 

• Does the crew support the system? 

• Do operations facilities exist? 

• What are the operational impacts? 

Schedule 

• How many critical paths can be identified? 

• What is the interval between the primary, secondary, and tertiary critical paths? 

• How complex is the schedule's critical path? 

• How much reserve exists in the schedule? 

• How vulnerable is the schedule to outside events? 

• What is the likelihood and severity of schedule slides? 

• What is the past performance in this area? 

• How much "learning time" exists between development and qualification tests and future 
need dates? 

• Has the variability of each task or activity in the schedule been characterized consistently? 
Cost 

• What was the past performance in this area? 

• How much reserve exists? 

• What is the likelihood/severity of any cost overruns? 

• How aggressive is the cost estimation? 

• How are cost uncertainties accounted for? 

Programmatic 

• Are the required personnel and skills available? 

• How stable and reliable are all involved subcontractors? 

• Are the requirements stable and well-defined? 

• Are sufficient development resources, such as computers and office space, available? 

• Does the program have adequate support from the public, government agencies and 
elected/appointed officials? 

• Is funding to an appropriate level likely to be a problem? 

Safety 

• What is the severity/intensity of the potential hazards? 

• Could the hazards potentially threaten the station or the lives of the crew? 

• Could the hazard cause a debilitating injury? 

• Could the hazard temporarily diminish the crew capacity? 
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• What is the predicted duration of exposure to the hazard? 

• What is the frequency of exposure to the hazard? 

• Are adequate hazard mitigation factors built into the design? 

Step 2: Assessment 

Process of Threat Assessment 

• Predict probability of occurrence 

• Predict magnitude of consequences if event occurs 

• Identify Root Causes 

• Assess iteratively using gross filter to eliminate threats with trivial consequences and/or 
probabilities 

Tools for Threat Assessment 

• Threat definition: assumptions, constraints, and groundrules 

• Physical measurement 

• Risk Scoring Method 

• Probability of occurrence 

• Consequence of occurrence 

• Logic networking 

• Requirements changes 

• Relative Ranking 

• Fishbone diagrams 

• Pareto Charts 

Step 3 : Analysis 
Process of Analysis 

• Generate abatement options that reduce probability and/or consequences 

• Analyze/ Assess option impacts 

• Select abatement options 

Tools for Risk Analysis 

• Trend analysis 

• Decision Analysis 

• Statistical process control 

• Variability histograms 

• Probabilistic network tools 

Step 4: Abatement 

• Develop a specific plan with goals, milestones, and personnel assignments 

• Measure progress 

• Determine a "fail-safe" point for engaging a "fallback" plan 

• Provide explicit closure criteria to define the end point 
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Tools for Risk Abatement 

• Flow diagrams 

• Abatement status and analysis 

• Closure criteria 

Roles 

• Serve as risk managers for teams 

• Ensure teams are trained in integrated risk management approach 

• Elevate risks to next higher team if necessary 

• Have primary risk identification responsibility 

• Participate with team in assessment, analysis, and abatement 

• Develops structured risk management process and tools 

• Supports total implementation of integrated risk management approach 

• Provides consulting on risk management 

Team Responsibilities 

• Identify threats to team objectives 

• Apply risk management process 

• Maintain list of "about 10" top team risks 

• Elevate risks as necessary 

Interdependencies 

• Communicate in all directions about how risk or abatement option might affect other 
elements 

• Status team above, monitor team below 

• Assign risk management point of contact 

• Notify any risk at stage level and above 

Your Biggest Role: Commitment 

• Help build an environment where risk is confronted as a challenge, not covered as a 
deficiency. 

• Remember the big picture: You may have the only eyes, ears, brain cells that will 
encounter a particular element of risk 

• Outline your action plan. 

• What are the first actions you will initiate as a result of the training? 

• What is your schedule? 

• When will your top "about 10" list be in place? 

• Whose help do you need? 

• How will you evaluate your results? 

The principal focus of most RM efforts is risk analysis and quantification It represents the 
step most frequently singled out for analytical and empirical treatment. As a result, the effort 
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associated with risk identification tends to be overshadowed by this emphasis when a more 
balanced approach would assure a higher confidence that the risk themselves have been identified 
prior to the risk analysis step. Formal risk analysis attempts to answer quantitatively questions 
associated with the range of consequences inherent in the performance of hardware and software 
developed to achieve a specific requirement and the interactions and effects of the human element 
wherever human interaction is required or likely. Risk analysts define risk as a combination of the 
probability of an undesirable event with the magnitude of each and every foreseeable 
consequence. These range from inconsequential to the catastrophic. One major problem with risk 
assessment is that it has sometimes been available only toward the end of the project, long after 
the design has been frozen and the system manufactured and tested. In this case, all that can be 
done with the results of the risk assessment analysis is to use it in support of the decision whether 
or not to go ahead with the final phase of the project. Except for indicating for retrofits, any 
other use is precluded, because no effort has been made to apply risk assessment in the planning 
and development phase of the project, where it would provide an additional component for the 
many decisions between technical alternatives. In its worst use, late risk assessment activity 
provides conclusions barely in time for the final decision on the fate of the project. In new 
systems, RM is coming to be accepted in engineering as a way of comparing the risks inherent in 
alternative designs, spotlighting the high-risk portions of a system, and pointing up techniques for 
mitigating those risks. For older systems, risk analyses conducted after they have been built and 
operated have often revealed crucial design faults. Risk analysts define risk as a combination of 
the probability of an undesirable event with the magnitude of each and every possible 
consequence. Reliability, while related to risk, is only part of the picture. This concept excludes 
both the consequences of failure and any causes that happen to be external to the component or 
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system. 


An attempt was made to compare pre-phase A and final reports of some successful projects 
such as GPPM, MAPS, LDEF, LITE, and HALOE in order to draw conclusions as to input 
parameters that make projects successful. Similarly, attempts were made to identify poor or 
unsuccessful projects in order to identify the input parameters (stated possibly in pre-phase A 
report) that may have contributed to poor performances. It was indeed very surprising to find out 
how little documentation was available at various potential sources such as the libraries and 
numerous offices. It proved to be a practically impossible task to find a set of “before” and 
“after” reports for each project. It also proved to be quite difficult to contact personnel about the 
above projects. It seems that most projects take long duration and it is common to have different 
personnel in various stages of a given project. Any project inherently contains a combination of 
the following risks: technological, performance, and price. NASA projects are more vulnerable to 
these risks because NASA needs much smaller quantities and most new items must qualified to 
endure much harsher environmental operating conditions. 

What is a successful or a failed Project? 

This issue has been pointed out as a major output parameter. This issue is not unique to NASA 
projects. A project may be over the budget and delayed, but the end product and the 
accomplishments may be very satisfactory. Then, it is difficult to judge project’s success while it 
is easier to identify poor projects. It is also hard to define what constitutes a failed project, but 
there appear to be some common some common aspects that suggest certain characteristics are 
strongly related to perceived project failure. There are three distinct aspects of project outcome 
that can be used as benchmarks against which to assess the success or failure of a project. These 
aspects are: 1) the implementation process itself; 2) the perceived value of the project; 3) client 
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satisfaction with delivered product. It is difficult to measure the performances of these aspects 
without bias. Perceived causes of project failure will vary, depending on which outcome measure 
is used to assess performance. 
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LITERATURE REVIEW 


This review was conducted with a clear requirement that general project management type 
publications would be excluded, as there are literally thousands of references in this area. This 
exclusion significantly reduced the number of references that could be listed at the end of this 
report. General topic of reliability was also excluded although it really is one of the foundations 
of any meaningful RM work. System reliability analysis, deterministic or probabilistic, is indeed a 
much larger area beyond the intent of this paper. Old Dominion University Library and NASA 
Langley Research Center Technical library resources were used. In addition, on-line 
FIRSTSEARCH database was utilized. Project risk assessment is a subset of overall project 
management field. The reference section contains 59 citations deemed relevant to this study. 
Many of these references cite other references, some of which are relevant. Such secondary 
references are not included in the list, as the reader should determine the relevancy of these 
references. The survey has found that many references lack any quantitative matter, and instead, 
provide lists of “do’s” and “do not’s”. Those that provide non-qualitative arguments are generally 
limited to re-visit of well-known and simple applications of probability, PERT, etc. Simulation is 
mentioned often, but rarely developed in an effective manner. More analytical applications such 
as reliability optimization or overall system reliability calculations are also rare and often limited to 
much overused drawings consisting of familiar boxes that represent a “system”. It is hard to 
believe that real problems can be solved with such simplistic applications, but such applications 
still help. 

References can be placed into four broad groups: 

1 . Directly related to RM, these are discussed further, 

2. NASA and DOE based support references [1 1,26,28,29,30,39,55,56], 
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3. Relevant academic references [2,12,13,14,19,31,32,35,38,42,46,51,54,59], 

4. Relevant practitioner references [1,9,10,33,37,45] 

The review revealed that a similar study was performed for NASA[5] ten years ago by another 
ASEE Fellow at Marshall Space Flight Center. Program Risk Analysis Handbook[6] is available 
here at NASA Langley Research Center Technical Library and no attempt will be made to 
reproduce the basic information presented in it. The reader is referred to pages 60 to 70 [6] 
(Bibliography Section) where 150 general risk related references are listed in nine categories. 
General Program Risk Analysis, Subjective Probability Encoding, Cost Risk Analysis, Decision 
Analysis, Network Methods, Technical/Performance Risk Analysis, Application/Case Studies, 
Group Consensus Methods, and Time-Cost Trade-Off Methods. These references are a mixture 

of academic articles and other sources such U.S. Army and Airforce technical reports. A portion 
of the references cited in above handbook[6] are applicable for the current effort, but none is 
listed in this report to avoid redundancy. The terms “risk” and “uncertainty” continue being used 
interchangeably in the literature in reference to project risk management. Among the numerous 
project risk management or risk assessment type publications, a few are unique to Aerospace area 
It is noteworthy that such publications do not offer any significant or special tools even though 
the authors are from industries similar to NASA’s areas. For example, references by Hamman et 
al. [25], Shaw [52], Hopkins [27], Vlay and Brekka[57], Balthazor [3], Kaplan[34], 
Billingham[6], Marcoux and Woop[36], Feiler and Geminder[18], and Giuntini and Storm[22] are 
all published by members of the aerospace industry. Each title sounds highly relevant and useful, 
but close examination shows that these papers are largely of re-packaged information easily 
available in established texts. These non-refereed conference papers provide little help for 
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effective RM work although the authors seem to believe otherwise. These and many other non- 
aerospace publications such as the one by Dey et al.[8] seem to have the following common 
points: some simplistic normal and/or Beta distribution based probability applications and a 
numerous flow charts which seem to go in several confusing directions. The search overall has 
been disappointing although it is possibly very complete. 

Reference [53] confirms this ASEE Fellow’s long standing confidence for risk analysis 
software @RISK [40], It is reported [53] that majority of practitioners use @RISK in project 
risk analysis and assessment in U.K. This excellent software is recommenced for use NASA if it 
is not already being used. Two risk analysis type publications by Sarper [47,50] also use @RISK 
software which simply performs Monte-Carlo simulation in a methodological way. This general 
risk analysis concept is a standard tool and is reported in numerous NASA and other references. 
Then, Monte-Carlo or static simulation is the prime tool for project risk analysis. The Handbook 
[6] lists a number of other simulation software available before 1987. 

The paper by Philipson[41] presents an application of risk analysis techniques in the area of 
safety assessment, in this case to determine the risk profiles associated with the possibility of 
space vehicle or missile launch accidents. Fragola and McFadden [20] move the reader into the 
system design area, starting from the examination of very standard reliability concepts applied to 
complex systems such as the Space Station. The paper by Preyssl[43] is an overview of the risk 
assessment philosophy that is being applied at the European Space Agency (ESA). The author 
presents a high level approach to risk assessment, which is part based on the tailored use of expert 
opinion to prioritize intervention on system safety issues. This approach is being applied to the 
design of the European module of the International Space Station effort. The theme of risk based 
prioritization and design is described, at a more detailed level, in the paper by Frank[21], His 
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emphasis is on the combined use of risk assessment and decision theory techniques, including 
multi-attribute prioritization and decision techniques. The reader should evaluate this approach as 
a proposal for how risk assessment could be used more extensively in support of design decisions, 
even when the subject of decision does not specifically consider safety matters. 


Incorporation of Risk in Optimization Projects 

Term “optimization” often has different interpretations within the engineering community. 
Optimization usually means that a mathematical model (objective function and constraints) is 
developed to represent the decision problem under consideration. It is not the intent here to 
review this major topic, but it is important to state that optimization models fall between the 
extremes of being totally linear and totally nonlinear. Most NASA and engineering decision 
problems would be generally non-linear. Although risk has long been attached to optimization 
models that are normally deterministic unless otherwise specified, many analysts continue to use 
deterministic coefficients both with linear and nonlinear models. Coefficients are sometimes 
adjusted with safety factors to account for randomness and the risks associated with such 
randomness, but this approach does not adequately account for the interrelationships among the 
various underlying random variables naturally found in any real decision making process. The 
field of engineering optimization is usually nonlinear, but deterministic. When randomness must 
be considered, the problem becomes a Stochastic Mathematical Model that may contain random 
coefficients in its objective function and/or constraints. This randomness is adequately described 
by various probability density functions, which may or may not exhibit some form of dependency 
relationships. There are three [48,49] distinct ways risky or stochastic optimization models are 
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classified: 


• “Here and Now” approach to find values for decision variables, 

• Chance-constrained [ ] programming that first defines acceptable risks of constraint violation 
and then converts the problem into its deterministic equivalent, 

• “Wait and See” approach to describe the probability distribution of the optimal value, 

In addition to general optimization, reliability optimization should include risk too, but this is 
rarely done [15], Two usual objectives of reliability optimization models are maximization of 
system reliability (subject to cost and other technical constraints) and minimization of overall 
system cost (subject to minimum reliability and other technical constraints). It is not really 
possible to precisely know the reliability of each component before they are built and tested. In 
addition, available budget and other resources may well be random too at early stages of design. 
Stochastic reliability optimization models can be used to describe the decision process. Any of 
the three distinct stochastic approaches above can be used. 
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FUTURE RESEARCH 


The original goal of the project was to come up with a function that can predict project 
success as a function of various inputs. In other words, is it possible or even meaningful to come 
with a measuring device or mathematical function that can, as output, determine the probability of 
success? Inputs would be quite diverse. Some inputs are funding level and availability, technical 
know how level, scheduling issues, desired and required project expectations such as orbital 
parameters and data quality requirements, among others. 

It is proposed that this research be undertaken with a potential NASA Langley Research 
Center grant. One possible format of grant could that of Graduate Student Researcher Program 
or GSRP. This author intends to submit a GSRP proposal for the next funding period. These 
applications are due in early February 1998 and would be evaluated by mid March 1998. If 
granted, award would be made for a period of one year (renewable to second year) beginning in 
fall 1998. A qualified graduate student (U.S. Citizen) will be nominated in the application. 
Although this award provides no funds or release time for the faculty, the student will be able to 
carry out this research with under the guidance of this faculty who has had the necessary exposure 
during this summer assignment. 
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CONCLUSION 


It is essential that NASA Langley Research Center establish a fully functional central 
repository where pre-phase A report along with its supporting trade studies and the final report of 
each project is stored. It is suggested that a computerized storage system (ASPER, already 
available in Systems Engineering Office) be utilized. Such a system would provide the historical 
data needed in making risk assessment analysis as a part of overall RM study of future projects. 

It should thus be clear that the applications presented is still the result of an environment that 
does not yet encourage the use of RM in as broad a range of problems as would be possible if the 
political and traditional obstacles had been fully removed. Program and project managers need to 
be convinced that RM is much broader than traditional safety applications. The time has come to 
begin to apply these techniques and risk-based design will indeed help maximize the return on 
investment as demanded by public at large. 

This study has shown that formal RM is still underutilized, for it is not without cost. 

However, its benefits have neither clearly been demonstrated nor fully appreciated. The best 
approach for RM should be one that has intense risk assessment activity at the very beginning of a 
project, in the planning and development phases, diminishing to a more sedate phase in the later 
phases of the project. This approach will put the main focus not only on identifying weak spots in 
the system. But also on another strong point of quantitative risk analysis, the ability to compare 
the risks of alternate approaches. Many of the most problematic uncertainties in the calculation of 
a risk component cancel when a ratio is formed, resulting in values precise enough for a 
meaningful comparison of various risks Thus, cost -benefit evaluation of alternatives could be 
supported at an early stage, helping to keep the risk at a cost-effective low level and yielding, at 
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the same time, a continuously updated and refined estimate for each component of the total risk. 
This approach would place risk assessment and management in its proper place as small, yet 
important role in the large set of activities that make up the project under consideration. At the 
same time, this approach avoids focusing the impact of risk assessment on the final phases of a 
project, after large investments of time, resources, and money have already been made and 
positions, often with political ramifications, have already been adopted. To implement this 
approach, project managers need to define the function of risk assessment in their projects, 
changing it from the evaluation of a finished product to the creation of an optimal deliverable 
product. In doing so, it should also be recognized that the necessary and worthwhile projects 
should not solely be driven by an excessive attention to risk, but by NASA and societal goals, 
needs, and concems-of which risk is only one facet. 

There is ample evidence that risk analysis would be worthwhile for any major project 
including NASA projects. It is; however, clear that risk analysis has sometimes been considered 
an unnecessary effort that is already accounted for in the main design stage. The literature 
indicates it is very important to conduct meaningful risk studies that go well beyond the fudge 
factor” addition that some claim is all that risk analysis does. This is a difficult stigma industrial 
engineers, systems engineers, and operations research analysts face quite often when dealing with 
other staff from purely technical areas. It is somewhat understandable why keenly design 
oriented staff may often prefer to avoid RM altogether. This research has shown that a 
significant amount of RM work is being pushed by those who also peddle other buzz words such 
as the Total Quality Management, Quality Circles, etc. Non-technical persons are attempting to 
tell rocket scientists what to do in their work. This is certainly not true in many cases when RM is 
advocated by other technical staff, but non-technical “know it all” and “fix it all up with TQM” 
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mentality is sometimes behind much RM push. There is considerable amount of RM publications 
written by non-technical persons whose goal are to advise scientists and engineers on how to do 
their jobs. This interference appears to cause some resentment among those who can most 
benefit from RM. 


41 



ACKNOWLEDGEMENTS 


I am very grateful for Mr. Motley’s assistance during my term here at NASA Langley 

Research Center. He was always kind and sincere in providing help and guidance for this project 

and my other needs. The staff at NASA Langley Research Center Technical Library are also 

thanked for excellent and cheerful service. 

REFERENCES 

1 . Alford, M., “Executable Model Support for Risk Management”, Proceedings of the Third 
Annual International Symposium of the National Council on Systems Engineering, 499-506, 
Arlington, VA, 1993. 

2. Bard, J.F et al. “ An Interactive Approach to R&D Project Selection and Termination”, IEEE 
Transactions on Engineering Management, 35, 139-146, 1988. 

3. Balthazor, L.R., “So You think You know Where You Are? A Review of Some Techniques 
Used in Evaluating and Predicting Development Schedules”, Proceedings of Royal 
Aeronautical Society, 105-133, London, U.K., 1987. 

4. Batson, R.G., “A Practitioner Oriented Survey of Program Risk Analysis Methodology”, Joint 
ORSA/TIMS National Meeting Presentation Document, Washington, D.C., 1988. 

5. Batson, R.G., “Risk Analysis Methodology Survey”, NASA Marshall Flight Center Report 
N88-15604, Huntsville, AL, 1987. 

6. Batson, R.G., Program Risk Analysis Handbook, NASA Marshall Flight Center TM-1003 11, 
Huntsville, AL, 1987. 

7. Bell, T.E., “Special Report-Managing Risk in Large Complex Systems”, IEEE Spectrum , 26, 
21-52, 1989 

8. Billingham, J., “Risk and Value Analysis of Search for Extraterrestrial Intelligence (SETI)”, 
37 th Congress of the International Astronautical Federation, Paper No. IAA-86-469, 
Innsbruck, Austria, 1986. 

9. Booth, S., “Risk Assessment and Mitigation Planning Early in the Development Life Cycle”, 
Proceedings of the Third Annual International Symposium of the National Council on 
Systems Engineering, 507-512, Arlington, VA 1993. 

10. Brekka, L.T. and Vlay, G.J., “Integrated Risk Management Procedures and Tools”, 


42 



Proceedings of the Third Annual International Symposium of the National Council on 
Systems Engineering, 491-498, Arlington, VA 1993. 

11. Brinkley, R. and Stone, D., “International Space Station Program Risk Management Team 
Execution Plan’, NASA-JSC, Houston, TX, July 1995. 

12. Buchan, D.H., “ Risk Analysis - Some Practical Suggestions”, Cost Engineering, 36, 29-34, 
1994. 

13. Chapman, C., “Project Risk Analysis and Management - The PRAM Generic”, International 
Journal of Project Management, 15, 273-281, 1997. 

14. Couillard, J., “The Role of Project Risk in Determining Project Management Approach”, 
Project Management Journal, 26, 3- 1 5, 1 995 . 

15. Decker, B. and Sarper, H., “Evaluation of Vendors When Reliability Data is Uncertain”, 
Proceedings of Engineering Design and Automation Conference, 1521-1524, Bangkok, 
Thailand, 1997. 

16. Dey, P. et al., “Planning for Project Control Through Risk Analysis: A Petroleum Pipeline- 
laying Project”, International Journal of Project Management, 12, 23-33, 1994. 

17. Feiler, A M. and Geminder, R., “Test Cost Savings Through Risk Analysis”, Twelfth 
Aerospace Testing Seminar, 25-28, Manhattan Beach, CA, 1990. 

18. Feiler, A.M. and Geminder, R., “Managing Project Technical, Cost, and Schedule Risks”, 
32nd Annual Technical Meeting of Institute of Environmental Sciences, 52-63, Dallas-Ft. 
Worth, TX, 1986. 

19. Finley, E D. and Fisher, D.J., “Project Risk Assessment Using Monte-Carlo Methods”, Cost 
Engineering, 36, 24-28, 1994. 

20. Fragola, J.R. and McFadden, R.H., “Synthesis of Failure Rates for Space Station External 
Orbital Replacement Replaceable Units”, Reliability Engineering and System Safety, 49, 
237-253. 

21. Frank, M.V., “ Choosing Among Safety Improvement Strategies: A Discussion with Example 
of Risk Assessment and Multi-Criteria Decision Approaches for NASA”, Reliability 
Engineering and Systems Safety” , 49, 311-324, 1995. 

22. Giuntini, R E. and Storm, RE, “Cost Effective Management of Space Venture Risks”, Ninth 
Aerospace Testing Seminar, 131-136, LA, CA 1985. 

23. Guerro, S. etal., “The Cassini Mission Risk Assessment Framework and Application 
Techniques”, Reliability Engineering & System Safety, 49, 293-302, 1995. 


43 



24. Haber, J.M., “Risk Assessment and Program Management”, Twelfth Aerospace Testing 
Seminar, 31-38, Manhattan Beach, CA, 1990. 

25. Hammon, C. et al., “Risk Assessment Preprocessor”, 19th Annual Department of Defense 
Cost Analysis Symposium Report No. AD-A161914, Leesburg, VA, 1985. 

26. Hoban, T. and Lawbaugh, W.M., Readings in Systems Engineering, NASA Report SP-6102, 
Washington, D C., 1993. 

27. Hopkins, D.K., “Planning of Risk in Defence Development Projects”, Proceedings of Royal 
Aeronautical Society Symposium , 76-104, London, U.K., 1987. 

28. Hoffman, E.J. and Lawbaugh, W.M (editors), Issues in NASA Program and Project 
Management, NASA Report SP-6101 (12), Washington, D C., 1997. 

29. Hoffman, E.J. and Lawbaugh, W.M. (editors). Issues in NASA Program and Project 
Management, NASA Report SP-6101 (11), Washington, D.C., 1996. 

30. Hoffman, E.J. (editor), Issues in NASA Program and Project Management, NASA Report SP- 
6101 (08), Washington, D C., 1994. 

3 1 . Hottenstein, M.O. and Dean, J.W., “Managing Risk in Advanced Manufacturing 
Technology”, California Management Review, 34, 112-126. 

32. Hulett, D.T., “Project Schedule Risk Assessment”, Project Management Journal, 26, 21-31, 
1995. 

33. Jones, J.H., “Evaluating Project Risk with Linguistic Variables”, Proceedings of the Fourth 
Annual International Symposium of the National Council on Systems Engineering, 979-986, 
San Jose, CA, 1994. 

34. Kaplan, S., “Safety Risk Assessment on the Space Station Freedom”, AIAA Space Programs 
and Technologies Conference, 7-11, Huntsville, AL, 1990. 

35. Klein, J.H. et al., “Project Risk Analysis based on Prototype Activities”, The Journal of the 
Operational Research Society, 45, 1994. 

36. Marcoux, J. and Woop, G., “Risk Assessment and Management Space European Standard - 
RAMSES”, 47th International Astronautical Congress, Beijing, PRC, 1996. 

37. May, C. and Olson, R., “Program Risk: The Balancing of Performance, Schedule, and Cost 
Risks”, Proceedings of the Fifth Annual International Council on Systems Engineering, 353- 
358, St. Louis, MO, 1995. 


44 



38. Mustafa, M.A. and Al-Bahar, J.F., “Project Risk Assessment Using the Analytical Hierarchy 
Process”, IEEE Transactions on Engineering Management, 38, 46-52, 1991. 

39. NASA Langley Research Center, Systems Engineering Handbook for in-house Space Flight 
Projects, LHB 7122.1, Hampton, VA, 1994. 

40. Palisade Corporation, @RISK, Risk Analysis and Modeling, Newfield, NY, 1996. 

41. Philipson, L.L., “Risk Profile Derivations for Space and Missile Launch Hazards”, Reliability 
Engineering and System Safety, 49, 2 1 7-23 6, 1 995 . 

42. Pinto, J.K. and Mantel, S.J., “The Causes of Project Failure”, IEEE Transactions on 
Engineering Management, 37, 269-276, 1990. 

43. Preyssl, C., “Safety Risk Assessment and Management- the ESA Approach”, Reliability 
Engineering and System Safety, 49, 303-309, 1995. 

44. Reed, W. and Woodhams, W., Executive and Deputy Executive Directors, Virginia 
Commercial Space Authority, Norfolk, VA Meeting Notes, July 8, 1997. 

45. Riggs, J.L., “Contract Risk Exposure”, Proceedings of the Third Annual International 
Symposium of the National Council on Systems Engineering, 519-524, Arlington, VA 1993. 

46. Rutgers, J. A. and Haley, H D, “. . .. Project Risks and Risk Allocation”, Cost Engineering, 
38, 27-30, 1996. 

47. Sarper, H., “Extreme Value Flight Duration Analysis of Four-Engine Spacecraft”, Journal of 
Spacecraft and Rockets, 34, 402-405, 1997. 

48. Sarper, H., “A Simulation Approach for the Analysis of the Effect of Randomness in Cutting 
Surface Constraints in Machining Economics Problem”, International Journal of Production 
Research, 33, 153-161, 1995. 

49. Sarper, H., “Monte-Carlo Simulation for Analysis of the Optimum Value in Stochastic 
Mathematical Programs”, Mathematics and Computers in Simulation , 35, 469-480, 1993 

50. Sarper, H , “@RISK and UNIFIT II: Powerful Tools for Engineering Economists”, HE 
Engineering Economy Division Newsletter, Spring 1993. 

51. Seiler, F.A., “On the Use of Risk Assessment in Project Management”, Risk Analysis, 10, 
365-366, 1990. 

52. Shaw, T.E., “ An Overview of Risk Management Techniques, Methods and Application”, 
Proceedings of AIAA Space Programs and Technologies, 1-14, Huntsville AL, 1990 


45 



53. Simister, S.J., “Usage and Benefits of Project Risk Analysis and Management”, International 
Journal of Project Management, 12, 5-8, 1994. 

54. Smith, L.A. and Mandakovic, T., “Estimating - The Input into Good Project Planning”, IEEE 
Transactions on Engineering Management, 32, 181-185, 1985. 

55. U S. Airforce, ASC Integrated Risk Management Guide: Draft Wright-Patterson AFB 
Aeronautical Systems Center, OH, 1994. 

56. U.S. Department of Defense, Transition from Development to Production... Solving the Risk 
Equation. DOD 4245. 7-M, 1985. 

57. Vlay, G.J. and Brekka, L.T., “Risk Management Integration with System Engineering and 
Program Management”, Space Programs and Technologies Conference, Huntsville, AL, 
1990. 

58. Welch, S.S., “The PASDE Project: Lessons Learned from a Successful ‘Faster, Better, 
Cheaper’ Flight Experiment”, ASEE / LARSS Lecture, NASA Langley Research Center, 
Hampton, VA, July 22, 1997. 

59. Williams, T.M., “ The Two-Dimensionality of Project Risk”, International Journal of Project 
Management, 14, 185-186, 1996. 


46 



